Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography

The annotated bibliography highlights work in the area of algorithmic test generation from formal speci cations with guaranteed fault coverage, i.e., fault model-driven test derivation. A fault model is understood as a triple, comprising a nite state speci cation, conformance relation and fault domain that is the set of possible implementations. The fault model can be specialized to Input/Output FSM, Labeled Transition System, or Input/Output Automaton and to a number of conformance relations such as FSM equivalence, reduction or quasiequivalence, trace inclusion or trace equivalence and others. The fault domain usually re ects test assumptions, as an example, it can be the universe of all possible I/O FSMs with a given number of states, a classical fault domain in FSM-based testing. A test suite is complete with respect to a given fault model when each implementation from the fault domain passes it if and only if the postulated conformance relation holds between the implementation and its speci cation. A complete test suite is said to provide fault coverage guarantee for a given fault model. Alur, R., Courcoubetis, C., and Yannakakis, M.: Distinguishing Tests for Nondeterministic and Probabilistic Machines. In: Proceedings of the 27 ACM Symposium on Theory of Computing (1995) 363-372 The complexity of state distinguishability problems for nondeterministic and probabilistic nite state machines is studied. Bhattacharyya, A.: Checking Experiments in Sequential Machines. John Wiley & Sons (1989) The book provides an overview of an early work on checking experiments and fault detection in sequential circuits. Bochmann, G. v., Das, A., Dssouli, R., Dubuc, M., Ghedamsi, A., and Luo, G.: Fault Models in Testing. In: Proceedings of IFIP TC6 Fourth International Workshop on Protocol Test Systems. North-Holland (1991) 17-30 The paper provides an inventory of various fault models used for speci cationbased hardware and software testing. Various types of speci cations are covered in the survey. Bochmann, G. v. and Petrenko, A.: Protocol Testing: Review of Methods and Relevance for Software Testing. In: Proceedings of ACM International Symposium on Software Testing and Analysis. Seattle USA (1994) 109-123 The paper reviews existing protocol testing methods, including methods for FSM and LTS-based test derivation for various fault models. Boroday, S. Yu.: Distinguishing tests for nondeterministic nite state machines. In: Proceedings of the 11 InternationalWorkshop on Testing of Communicating Systems (IWTCS'98). Russia (1998) 101-107 The following fault model is considered. The speci cation is a completely de ned nondeterministic FSM. The conformance relation is the reduction relation. The fault domain is the set of all completely de ned deterministic FSMs each of which is a reduction either of the speci cation machine or of another auxiliary completely de ned nondeterministic FSM. No upper bound on the number of states in deterministic FSMs is assumed, a nite test suite complete for this fault model does not always exist. A necessary and su cient condition for its existence and a method for deriving a minimal adaptive test suite are presented. Boroday, S. Yu.: Simple Fault Checking For Automata Generated By A Fault Function. Cybernetics and System Analysis. Plenum Publishing New York. Vol. 31, No. 6. (1995) 835-841 It presents a method for deriving a test suite complete with respect to the fault model that includes a completely de ned deterministic I/O FSM, equivalence relation between such machines and fault domains comprising deterministic submachines of a given nondeterministic FSM (a fault function as in the FF-method) such that faults a ect either a single state or single transition without increasing the number of states in implementations. Chow, T. S.: Testing Software Design Modeled by Finite-State Machines. IEEE Transactions on Software Engineering. Vol. SE-4, No. 3. (1978) 178-187 The method of Vasilevskii is detailed and slightly modi ed allowing for arbitrary (and not necessarily su x-closed) characterization sets. It became known as the W-method. Friedman, A. D., and Menon, P. R.: Fault Detection in Digital Circuits. PrenticeHall (1971) Chapter 3 explains checking experiments on minimal completely de ned deterministic I/O FSMs. Fujiwara, S., Bochmann, G. v., Khendek, F., Amalou, M., and Ghedamsi, A.: Test Selection Based on Finite State Models. IEEE Transactions on Software Engineering. Vol. SE-17, No. 6. (1991) 591-603 The method of Vasilevskii (the W-method) is improved to the Wp-method. In this method, the transition checking phase relies on subsets (state identi ers) of the characterization set W used in the state checking phase (as in the W-method). Gill, A.: Introduction to the theory of nite-state machines. Mc Graw-Hill. New York (1962) One of the rst books where FSM-based testing is explained in detail. Gonenc, G.: A Method for the Design of Fault Detection Experiments. IEEE Transactions on Computers. Vol. C-19. June (1970) 551-558 The Hennie's approach is further elaborated in a more formal way for the class of FSMs possessing distinguishing sequences. The method became known as the D-method. Grunsky, I. S., and Petrenko, A.: Design of Checking Experiments with Automata Describing Protocols. Automatic Control and Computer Sciences. Allerton Press Inc. USA. No. 4 (1988) The problem of deriving tests with complete coverage of restricted faults in implementations of a completely de ned deterministic I/O FSM is rst formulated assuming that each implementation FSM is equivalent to a deterministic submachine of a nondeterministic FSM (called a fault function). It is demonstrated that the fault function generalizes other existing FSM fault models. A method for deriving a test suite with complete fault coverage is proposed. Grunsky, I. S.: Testing of Automata: from Experiments to Representations by Means of Fragments. In: Proceedings of the 11 International Workshop on Testing of Communicating Systems (IWTCS'98). Russia (1998) 3-14. An overview of several important results in FSM testing published in the Russian literature, some of the referred papers are not available in English. Hennie, F. C.: Fault Detecting Experiments for Sequential Circuits. In: Proceedings of the IEEE 5 Annual Symposium on Switching Circuits Theory and Logical Design. Princeton (1964) 95-110 Fundamental principles of the transition checking approach are rst enunciated. The speci cation is assumed to be a minimal, deterministic, completely de ned and strongly connected I/O FSM. The basic procedure yields a single test (checking experiment) for machines that have a distinguishing sequence provided that faults do not increase the number of states in implementations. Extended (though presented informally) procedures allow faults to double the number of states and apply to machines with two or more characterizing sequences (instead of a distinguishing sequence). Holzmann, G. J.: Design and Validation of Computer Protocols. Prentice Hall (1991) Chapter 9 is about conformance testing. Hsieh, E. P.: Checking Experiments for Sequential Machines. IEEE Transactions on Computers. Vol. C-20, No. 10. (1971) 1152-1166. The transition checking approach (pioneered by Hennie) is further elaborated for a special class of FSMs with Simple I/O sequences. The latter were later rediscovered under the name of UIO sequences. Kohavi, Z.: Switching and Finite Automata Theory. McGraw-Hill Computer Science Series. New York (1970) Chapter 13 is about state-identi cation and fault-detection experiments. Minimal completely de ned deterministic I/O FSMs are considered. Koufareva, I., Petrenko, A., and Yevtushenko, N.: Test Generation Driven by User-de ned Fault models. In: Proceedings of the 12 International Workshop on Testing of Communicating Systems (IWTCS'99). Hungary (1999) 215-233 A method is proposed for deriving a test suite complete with respect to the following fault model: a completely de ned deterministic I/O FSM, equivalence relation between such machines and fault domains comprising all deterministic submachines of a given nondeterministic FSM (a fault function). The latter represents faults de ned by the user. Faults can increase the number of states in implementations. Lee, D., and Yannakakis, M.: Testing Finite-State Machines: State Identi cation and Veri cation. IEEE Transactions of Computers. Vol. 43, No. 3 (1994) 306-320 The complexity of deriving distinguishing and Simple (UIO) sequences from a completely de ned FSM is studied. E cient algorithms are presented. Lee, D., and Yannakakis, M.: Principles and Methods of Testing Finite-State Machines-A survey. Proceedings of the IEEE. Vol. 84, No. 8. (1996) 1090-1123 The paper contains an analysis of testing problems and existing solutions based on minimal completely de ned nondeterministic I/O FSMs. Some extensions to the basic framework are also discussed. It includes an extended list of references. Lukyanov, B. D.: Distinguishing and Control Experiments with Nondeterministic Automata. Cybernetics and System Analysis. Plenum Publishing. New York. Vol. 31, No 5. (1995) 691-696 Adaptive and preset experiments with nondeterministic I/O FSM are analyzed. Luo, G., Bochmann, G. v., and Petrenko, A.: Test Selection based on Communicating Nondeterministic Finite State Machines using a Generalized Wp-Method. IEEE Transactions on Software Engineering. Vol. SE-20, No. 2. (1994) 149-162 The fault model considered in this paper includes a minimal completely de ned nondeterministic I/O FSM, the equivalence relation between FSMs and the universe of all nondeterministic FSMs with a given number of states. It is demonstrated that the Wp-method can be extended to derive a test suite complete for this fault model. Luo, G., Petrenko, A., and Bochmann, G. v.: Selecting Test Sequences for Partially Speci ed Nondeterministic Finite State Machines. In: Proceedings of the IFIP Seventh International Workshop on Protocol Test Systems. Japan (1994) 95-110 The fault model considered in this paper includes a partially de ned nondeterministic I/O FSM, the quasi-equivalence (weak conformance) relation between FSMs and the universe of all completely de ned nondeterministic FSMs with a given number of states. The so-called HSI-method that is based on harmonized state identi ers, subsets of a characterization set, rst proposed for partially de ned deterministic FSMs, yields a test suite complete for this fault model. Moore, E. F.: Gedanken Experiments on Sequential Machines. In: Automata Studies. Princeton University Press. Princeton New Jersey (1956) 129-153 It is one of most referred papers in black box testing. A conceptual framework for FSM-based testing is proposed. The notions of simple and multiple checking (black box) experiments are introduced. The proposed approach for deriving a checking experiment requires the explicit enumeration of all FSMs with a given number of states. The resulting experiments allow one not only to detect a fault but also to locate it. Machine identi cation is thus achieved. Naito, S., and Tsunoyama, M.: Fault Detection for Sequential Machines by Transition-Tours. In: Proceedings of the IEEE International Symposium on Fault Tolerant Computer Systems (1981) 238-243 A transition tour gives a test sequence that covers every transition of a deterministic FSM and is complete with respect to all output (but not transfer) faults. The method became known as the T-method. Petrenko, A.: Checking Experiments with Protocol Machines. In: Proceedings of IFIP Fourth International Workshop on Protocol Test Systems. North-Holland (1991) 83-94 The paper provides a summary of results in the FSM-based testing previously obtained with the author's participation in the ex-USSR. Petrenko, A., and Yevtushenko, N.: Test Suite Generation for a FSM with a Given Type of Implementation Errors. In: Proceedings of IFIP 12 International Symposium on Protocol Speci cation, Testing, and Veri cation. USA (1992) 229243 It presents a method for deriving a test suite complete with respect to the fault model that includes a completely de ned deterministic I/O FSM, equivalence relation between such machines and fault domains comprising all deterministic submachines of a given nondeterministic FSM (called a fault function-based method, the FF-method). The latter represents faults de ned by the user. It is assumed that no fault increases the number of states in implementations. Petrenko, A., Bochmann, G. v., and Dssouli, R.: Conformance Relation and Test Derivation. In: Proceedings of IFIP Fifth International Workshop on Protocol Test Systems, 1993. North-Holland (1994) 157-178 The paper contains a survey of FSM-based test derivation methods with complete fault coverage. The idea of encoding an LTS by an I/O FSM in failure semantics is rst presented. The notion of multiple checking experiments becomes thus applicable to the LTS model and bridge between the FSM and LTS-based testing is established. Petrenko, A., Yevtushenko, N., Lebedev, A., and Das, A.: Nondeterministic State Machines in Protocol Conformance Testing. In: Proceedings of IFIP Fifth International Workshop on Protocol Test Systems, 1993. North-Holland (1994) 363-378 The paper present a method for deriving tests complete with respect to the following fault model. A completely de ned nondeterministic I/O FSM from a special class, the reduction relation between FSMs and the universe of all completely dened deterministic FSMs with a given number of states. The paper also contains preliminary results for testing FSM in context. Petrenko, A., Yevtushenko, N., Bochmann, G. v., and Dssouli, R.: Testing in Context: Framework and Test Derivation. Computer Communications (special issue on protocol engineering). 19 (1996) 1236-1249 A framework for testing FSM in context is presented. It is demonstrated that the problem can be reduced to testing in isolation w.r.t. the fault model that includes a partially de ned nondeterministic I/O FSM, the reduction relation between FSMs and the universe of all completely de ned deterministic FSMs with a given number of states. Petrenko, A., Yevtushenko, N., and Bochmann, G. v.: Fault Models for Testing in Context. In: Proceedings of IFIP Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols, and Protocol Speci cation, Testing, and Veri cation. Germany (1996) 163-178 The paper proposes various fault models appropriate for test generation with complete fault coverage when the implementation is embedded in a context and studies the relationships between them. Petrenko, A., Yevtushenko, N., and Bochmann, G. v.: Testing Deterministic Implementations from Nondeterministic FSM Speci cations. In: Proceedings of the 9 International Workshop on Testing of Communicating Systems. Germany (1996) 125-140 The following fault model is considered: a completely or partially de ned nondeterministic I/O FSM, the reduction relation between FSMs and the universe of all completely de ned deterministic FSMs with a given number of states. The proposed method yields a test suite complete for this fault model. Petrenko, A., Bochmann, G. v., and Yao, M.: On Fault Coverage of Tests for Finite State Speci cations. Computer Networks and ISDN Systems (special issue on protocol testing). 29, December (1996) 81-106 The paper analyzes the existing techniques for fault coverage of a given test suite for nite state models. Petrenko, A., and Yevtushenko, N.: Fault Detection in Embedded Components. In: Proceedings of 10 International Workshop on Testing of Communicating Systems. Korea (1997) 272-287 An e cient method is proposed for reducing the problem of testing in context to testing in isolation w.r.t. the following fault model: a partially de ned nondeterministic I/O FSM, the reduction relation between FSMs and the universe of all completely de ned deterministic FSMs with a given number of states. Poage, J. F., and McCluskey, Jr., E. J.: Derivation of Optimal Test Sequences for Sequential Machines. In: Proceedings of the IEEE 5 Symposium on Switching Circuits Theory and Logical Design (1964) 121-132 The speci cation is a completely de ned deterministic I/O FSM (Mealy machine). Faults are represented by a set of deterministic I/O FSMs. The latter are explicitly constructed from a given (small) set of structural faults in a sequential circuit. A method for deriving tests with complete fault coverage is proposed based on a notion of a product of deterministic FSMs. Rezaki, A., and Ural. H.: Construction of Checking Sequences Based on Characterization Sets. Computer Communications. Vol. 18, No. 12 (1995) 911-920 The Hennie's approach is further elaborated in a more formal way. The relationship between a single and multiple checking experiments is established. Sidhu, D. P., and Leung, T. K.: Formal Methods for Protocol Testing: A Detailed Study. IEEE Transactions on Software Engineering. Vol. SE-15, No. 4. (1989) 413-426 The paper reviews several basic FSM-based test derivation methods and presents a case study of experimental estimation of tests for their fault coverage. Note that Theorem 3 holds only under certain assumptions. Starke, P. H.: Abstract Automata. North-Holland/American Elsevier. (1972) It contains among other interesting things the theory of nondeterministic I/O FSMs used for testing. Tan, Q. M., Petrenko, A., and Bochmann, G. v.: Modeling Basic LOTOS by FSMs for Conformance Testing. In: Proceedings of the 15 International IFIP Symposium on Protocol Speci cation, Testing and Veri cation. Poland (1995) 123-138 The idea of modeling an LTS by an I/O FSM is further elaborated for trace semantics and failure semantics. It is demonstrated that the proposed transformations preserve the relations used for testing. Thus, test suites for the LTS model with complete fault coverage can be derived via detour to the FSM model. Tan, Q. M., Petrenko, A., and Bochmann, G. v.: A Framework for Conformance Testing of Systems Communicating through Rendezvous. In: Proceedings of the 26 IEEE International Symposium on Fault-Tolerant Computing. Japan (1996) 230-238 Several fault models for the LTS model are considered. For a given LTS speci cation a fault domain includes the universe of all LTSs with the given action set and the number of states not exceeding a given bound, the conformance relation can be one of the following. Trace inclusion, trace equivalence, failure reduction, failure equivalence, or nondeterminism reduction. The upper bounds for the complexity of tests complete w.r.t. each of these fault models are established. Tan, Q. M., Petrenko, A., and Bochmann, G. v.: Checking Experiments with Labeled Transition Systems for Trace Equivalence. In: Proceedings of the 10 International Workshop on Testing of Communicating Systems. Korea (1997) 167-182 The fault model includes an LTS, trace equivalence and the universe of all LTSs with the given action set and the number of states not exceeding a given bound. The analogues of the W-, Wp-, HIS(Harmonized State Identi cation) methods originally developed for the I/O FSM model are proposed to generate test suites complete w.r.t. this fault model directly from the LTS. Tan, Q. M., Petrenko, A.: Test Generation for Speci cations Modeled by Input/Output Automata. In: Proceedings of the 11 International Workshop on Testing of Communicating Systems. Russia (1998) 83-99 The fault model includes a reduced input-enabled and transition-deterministic I/O Automaton (I/O Transition System), trace equivalence and the set of all such automata with a given number of states. A method is proposed to generate a test suite complete with respect to this fault model. The method is an analogue of the FSM-based HSI-method. Trakhtenbrot, B. A., Barzdin, Y. M.: Finite Automata, Behaviour and Synthesis. North-Holland (1973) The statistical properties of FSMs, including ones related to testing, are studied, among other topics. Ural, H.: Formal Methods for Test Sequence Generation. Computer Communications. Vol. 15, No. 5. (1992) 311-325 The paper discusses several basic I/O FSM-based test derivation methods and some variations of them. Vasilevskii, M. P.: Failure Diagnosis of Automata. Cybernetics. Plenum Publishing Corporation. New York No. 4 (1973) 653-665 A method for deriving a complete test suite from a minimal completely de ned deterministic I/O FSM is proposed. The number of states in the implementations is assumed not to exceed a given bound. The method was later detailed by Chow and became known as the W-method. Polynomial upper and lower bounds on the length of tests are proved. Yannakakis, M., and Lee, D.: Testing Finite-State Machines: Fault Detection. Journal of Computer and System Sciences. 50 (1995) 209-227 Randomized algorithms for deriving a checking sequence (single experiment) from a reduced FSM are proposed. Yao, M., Petrenko, A., and Bochmann, G. v.: Conformance Testing of Protocol Machines without Reset. In: Proceedings of the 13 IFIP Symposium on Protocol Speci cation, Testing and Veri cation. Belgium (1993) 241253 The transition checking approach (pioneered by Hennie) is further improved for FSMs with Simple I/O (UIO) sequences. Yevtushenko, N., and Petrenko, A.: Synthesis of Test Experiments in Some Classes of Automata. Automatic Control and Computer Sciences. Allerton Press Inc. USA No. 4 (1990) Harmonized State Identi ers (HSI) are rst proposed as an alternative to a characterization set W. State identi ers are said to be harmonized if any two identi ers contains a common pre x that tells apart the corresponding states. Replacing the W set in the W-method by HSI gives the HSI-method for a reduced deterministic I/O FSM. This method uni es test derivation for both completely and partially de ned machines. An extension of this method for a subclass of nondeterministic I/O FSM is also suggested. Yevtushenko, N., and Petrenko, A.: Method of Constructing a Test Experiment for an Arbitrary Deterministic Automaton. Automatic Control and Computer Sciences. Allerton Press Inc. USA. No. 5 (1990) The paper considers the fault model that includes a deterministic I/O FSM possibly partially de ned and unreduced, the quasi-equivalence (weak conformance) relation and the universe of all completely de ned I/O FSMs with a given number of states. It presents the rst known method for deriving a test suite complete with respect to this fault model.